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Background / Context: 

The tutoring component of the Investing in Innovations (i3) funded Milwaukee 
Community Literacy Project (MCLP) is based on the Reading Recovery program. Reading 
Recovery focuses on in-school tutoring with lesson plans written, and assessments analyzed, by 
the tutors themselves. In a comparison of early literacy intervention programs, Pinnell et al found 
that Reading Recovery subjects performed significantly better than any other treatment and 
comparison group on all measures (Pinnell, Lyons, DeFord, Bryk, & Seltzer, 1994). 

The MCLP also contains a strong family engagement component. Involving families in 
tutoring programs can improve children’s academic knowledge, skills and confidence (Bryan, 
2005; Harvard Family Research Project, 2009). Encouraging family involvement in educational 
programs traditionally focuses on families attending events, receiving information from staff, 
volunteering (Epstein, 2001), and generally exhibiting “good parent” behaviors (Li, 2010). 
Getting to know families and the ways that their lives are structured outside of the educational 
setting may lead to a reciprocal relationship that can increase involvement (Graue & Hawkins, 
2010). The family component of a program is not only to make families aware of the program’s 
mission and goals but also to empower families in their child’s learning both in the program and 
at home. 

Purpose / Objective / Research Question / Focus of Study: 

The objective of this study is to measure the impact of the MCLP on reading achievement in two 
cohorts of participants. One cohort participated for two years in the 2011-2012 and 2012-2013 
school years. The other began participation in the 2013-2014 school year and is continuing to 
participate in 2014-2015. 

Setting: 

The MCLP is a collaboration between the Boys & Girls Clubs of Greater Milwaukee and 
the Milwaukee Public Schools (MPS) to provide one-on-one tutoring and family engagement to 
kindergarten through 3rd grade students in seven predominantly low-income and minority 
elementary schools. 

MPS, a district serving over 80,000 students, faces a significant challenge to teach its 
students how to read and write. According to the Wisconsin Knowledge and Concepts 
Examination (WKCE) only 15% of MPS students were proficient in reading (2011). This is in 
comparison to 35% statewide. The results of the WKCE are consistent with results of the 
National Assessment of Educational Progress (NAEP) and the ACT, which show that MPS 
students struggle with literacy throughout their education. 15% of 4th grade MPS students are 
proficient in reading (NAEP, 2011) while 14% of MPS 1 1th graders scored at least 21 on the 
ACT Reading Test, the benchmark identified for college readiness (Independent Analysis). 

The results of the NAEP further show that there are significant achievement gaps for minority 
and low-income students. 39% of 4th grade. White MPS students are proficient in reading, 
compared to 7% of Black and 15% of Hispanic students. 7% of 4th grade low-income 
(free/reduced lunch participants) MPS students are proficient in reading, compared to 48% of 
non-low-income students. These data demonstrate that the need for increased literacy 
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opportunities in the Milwaukee area is urgent, and that this need is even more pronounced for 
low-income and minority students. 


Population / Participants / Subjects: 

Cohort 1: 

A total of 496 students consented to participate in the first cohort. Of these, 245 were 
selected to participate in the MCLP and 251 were selected as control students. Of the 496 
students, 165 were in kindergarten, 177 in first, and 154 in second. 480 were eligible for free or 
reduced lunch, 404 were (81%) African American, and 57 (12%) Hispanic. 

Attrition was a problem during the two years that students participated. 223 students were 
excluded from the final analysis due to attrition, which represents a 45.0% overall attrition rate. 
However, it is important to note that students were excluded for exogenous reasons, like not 
taking the pre-test (5 students), moving away (186), being identified for a reading disability (30), 
and not taking the post-test (2). Regarding differential attrition, 107 (43.7) of participants and 
116 (46.2%) of control students dropped out. The 2.5% differential attrition rate, along with the 
45.0% overall attrition, and the exogenous nature of why students were dropped, suggests that 
the internal validity of the cohort 1 evaluation remained intact. 

Cohort 2: 

A total of 576 students across seven schools consented to participate in the second MCLP 
cohort. 286 students were randomly selected as MCLP participants and 290 as control students. 
Of the 576 students, 205 are in kindergarten, 214 in first, and 157 in second. 549 (95%) are 
eligible for free or reduced lunch, 459 (80%) African American, 71 (12%) Hispanic, and 51 (9%) 
have an lEP for speech/language. During the first year of participation, only 47 (8%) students 
dropped, all because they moved to another school. This included 20 (7%) participants and 27 
(9%) control students. 

Intervention / Program / Practice: 

Students are tutored during the school year for 30 minutes, up to three times per week, for 
two years. At each site, a program manager, who is also a certified teacher, oversees the tutors. 
There is a set lesson plan template for tutors and students to follow during each tutoring session. 
After this activity, the tutor administers a running record assessment (this occurs every third 
lesson). This is followed by phonics activities. Next, the student reads a book that is at his/her 
instructional level and then does a writing activity. Finally, the tutor reads a portion of a book 
aloud. 

All students are assessed with the PALS (Phonological Awareness Literacy Screening) at 
the beginning of the school year. This assessment is used to determine each student’s needs and 
help create individual lesson plans. Tutors complete all-program trainings at the beginning of the 
year, as well as follow-up training several times throughout the year. In addition, they receive 
training at their sites given by program managers. Tutors are observed at least monthly by 
program managers and receive feedback following these observations. 

Each site has a parent partner who works with each participating student’s family. Their 
work is designed to bridge the divide between school and home by translating literacy concepts, 
educating families about a variety of literacy activities, and validating the literacy practices 
already happening in the home. Parent partners help families see how they already are 
incorporating literacy into their children’s lives and show parents how to promote literacy more 
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effectively. Parent partners stay connected with families through a monthly newsletter, monthly 
family events at each site, and phone calls or emails. These communications are designed to keep 
families aware of student progress in MCLP, help families promote literacy at home, and address 
any attendance issues that arise during the program. Parent partners also conduct home visits for 
all students twice during the summer between their first and second year of participation. These 
visits are viewed as opportunities to connect with the family in their own space and learn about 
the literacy activities already taking place in the home. 

Research Design: 

The evaluation utilizes a randomized control trial to isolate the impact of MCLP on 
reading achievement. A random selection of kindergarten, 1st, and 2nd grade students in six 
MPS schools were selected to participate in Cohort 1 during the 2011-2012 school year. A 
second cohort was selected to participate in seven schools during the 2013-2014 school year. 
Students were randomly selected in the fall of each year and stratified by school and grade level 
within school. Students with a reading-related lEP or who were English Language Learners, 
were not eligible for participation in the evaluation but were eligible to receive tutoring. 

The primary outcome used to evaluate the MCLP is the Measures of Academic Progress 
(MAP) reading assessment published by the Northwest Evaluation Association (NWEA). The 
MAP is an on-line, adaptive assessment that has been shown to be both valid and reliable. MPS 
administers the MAP to all students three times each year, in the fall, winter, and spring. As 
such, to measure the impact of the MCLP on the first cohort, the evaluation was able to use the 
fall of 201 1 MAP results as the pre-test and the spring of 2013 results as the post-test. To 
measure the impact of the MCLP on the second cohort, the evaluation uses the fall of 2013 MAP 
results as the baseline and will use the spring of 2015 results as the post- test. The spring of 2014 
MAP results are used to measure interim results. 

MPS uses the MAP for Primary Grades (MPG) for kindergarten, 1st grade, and 2nd grade 
students. 3rd grade students take the MAP. The MPG is vertically aligned to the MAP, so that 
the same score on both assessments suggests the same reading achievement level. The main 
difference between the two is that the MPG includes auditory technology to help students 
complete the assessment. 

The statistical analysis of the MCLP follows an Intent to Treat Model, where students 
selected to participate in the MCLP are included in the analysis regardless of how much tutoring 
they received. This is done to maximize the internal validity of the study. However, due to 
missing data, both participant and control students who moved away or were identified as having 
a disability that prevented them from receiving literacy tutoring were excluded from the analysis. 

Data Collection and Analysis: 

The project has a data sharing agreement with the Milwaukee Public Schools (MPS) that 
allows it to access achievement results (MPG, MAP, PALS) and student demographics. The 
project itself maintains participation records and fidelity of implementation results. 

To analyze the impact of the MCLP, the evaluation used separate generalized linear 
statistical models with robust standard error estimators to compare the reading achievement 
growth of participants and controls for kindergarten, first, and second grade students. The results 
of the three models were then pooled to estimate the overall impact of the MCLP for each cohort. 
Post reading achievement scores were standardized to improve interpretability. 
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For cohort 1, all three grade-level models predicting post MAP/MPG scores controlled 
for the separate fixed interactions of school effects with both baseline fall 2011 MAP/MPG 
reading and MAP/MPG math results. The fixed effects of gender, race, disability status, and 
free/reduced lunch eligibility were also tested in the model. Ultimately, only the main effect of 
race was found to uniquely predict MAP/MPG reading results and was included in the model. 
The rest were excluded because they were not found to uniquely predict post-test reading 
achievement. 

For cohort 2, all three grade-level models predicting post MPG scores controlled for the 
fixed interaction of baseline MPG reading and school effects. The kindergarten and first grade 
models also controlled for the fall results of the Phonological Awareness Literacy Screener 
(PALS), used for the first time by MPS with its kindergarten and first grade students. MPG math 
baseline scores was considered as a covariate, but its inclusion in the kindergarten and first grade 
models did not reduce standard errors nor improve model fit, so it was not included. However, 
MPG math baseline scores did improve model fit and reduce standard errors for second grade 
students, so was included in that model. Spring 2014 PALS results were also tested for first and 
second grade students using the same model. 

Findings / Results: 

After two years of participation, the MCLP was found to have a small but significant 
impact on the reading achievement of the first cohort (0.12 standard deviations) (Table 1). 
However after only one year of participation, the MCLP was found to have a similar impact on 
the MAP/MPG scores of the second cohort (0.12 standard deviations) (Table 2). Second cohort 
MCLP participants also scored higher on the PALS; scores were .438 and .372 standardized units 
higher than control students (p < .0001). Pooled together, these results indicate that the MCLP 
had a large impact (.40 standardized effect) on spring PALS scores (p < .0001). 

Conclusions: 

The results of the evaluation of the MCLP are promising. Tested with rigorous evaluation 
methods, a randomized-control framework, the MCLP was found to have a statistically 
significant impact on the reading development of participants. Although attrition was a problem 
for the first cohort, there is no evidence that it affected the internal validity of these findings. 
There is also evidence that changes to the delivery of the MCLP from the first to the second 
cohort, improved its effectiveness. Specifically, the program revised its approach to working 
with kindergarten students. 

Taken together, these results suggest that the MCLP is a promising program for helping 
to address the serious challenge facing the Milwaukee Public Schools of teaching students to 
read. 
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Table 1: MCLP Cohort Final MAP/MPG Results 



Standardized 

Effect 

Robust 

Standard 

Errors 

P- 

value 

MCLP Kindergarten 

0.012 

0.123 


MCLP First 

0.118 

0.143 


MCLP Second 

0.288 

0.138 


Overall Impact (Weighted Pooled Results) 

0.122 

0.061 

<.05 

Table 2: MCLP 2^^ Cohort Interim MPG Results 



Robust 



Standardized 

Standard 

P- 


Effect 

Errors 

value 

MCLP Kindergarten 

0.117 

0.0968 

0.227 

MCLP First 

0.177 

0.0641 

0.006 

MCLP Second 

-0.047 

0.1114 

0.664 

Overall Impact (Weighted Pooled Results) 

0.12 

0.0482 

<.01 

Table 3. MCLP 2^° Cohort Interim PALS Results 



Robust 



Standardized 

Standard 



Effect 

Errors 

p-value 

MCLP Kindergarten 

0.438 

0.0895 

<0.0001 

MCLP First 

0.372 

0.0775 

<0.0001 

Overall Impact (Weighted Pooled Results) 

0.400 

0.0443 

<.0001 
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